Introduction


While testing a system, usability practitioners search for ways to improve the experience of the system. They constantly hunt for issues, problems or pain points that degrade the experience of the system. The main goal of usability testing is to explore ‘what is causing the problem?’ and also ‘why is it causing the problem?’. In order to answer these questions, it is imperative to first understand what usability is before we define usability problem.

(Rubin, & Chisnell, 2008) Usability is a quality of a product or service that allows the user to do what they want to do, in the way they expect to be able to do it, without hindrance, hesitation, or questions. A product or system is usable if it is useful (enables users to achieve their goals), efficient (enables users to achieve their goals with speed and accuracy), effective (behaves in way users expect it to), satisfying (projects positive perception, and feeling), learnable (easy and fast to learn or relearn product usage), and accessible (usable by people with disabilities). Thus, a usability problem would be an issue that negatively affects the user on any of the usability attributes mentioned above.

Rubin and Chisnell’s description of usability and Wilson’s description of usability problem goes hand in hand. Rubin and Chisnell, in their book ‘Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests’, states that usability is ‘invisible’. Wilson, at the UPA Idea Market, described usability problem as ‘elusive’. The descriptions are co-related. If something is invisible, it would indeed be difficult to find problems associated with it.

My definition of usability problem is as follows: ‘Usability problem is a problem, that when encountered by a user, induces the feeling of being lost, confused, nervous, frustrated, anxious, or unproductive’.



Categorization of usability problems


Practitioners categorize or organize usability problem is many ways. Few of them are discussed in the following sections


1. Dumas and Redish (1999) model


According to Dumas and Redish (1999), the most coherent way of organizing a problem is by means of its importance which has two aspects associated with it: scope (how widespread is the problem?) and severity (how critical is the problem?). The scope is further defined by the dichotomy between global (often the most important) and local (restricted to one screen/page/frame). Scope is not enough to organize problems in the order of priority. It needs to be combined with the severity of the problem. Dumas and Redish (1999) used a scale with 4 levels to roughly define the severity of the problems. Below list shows the four levels and its description (Dumas, & Redish, 1999).

Level 1 - Prevents the user from completing a task.

Level 2 - Creates a significant delay task completion.

Level 3 - Has a minor effect on usability.

Level 4 - Subtle with least effect.


Strength and Weakness

It is important to note that some problems might be very critical but restricted to a single page whereas some problems are repeated on multiple pages but aren’t critical at all. In such situation, the above-described model plays a very good role in categorizing the problems as it combines the scope and severity of a problem. On the other hand, there are few weaknesses associated with the model too. Firstly, a novice practitioner might not be able to recognize a number of co-related local problems as a global problem. Secondly, there are higher chances that a problem might go undetected as the model does not take into consideration design factors (heuristics) like the ease of use, intuitiveness, interaction, and so on.


2. Nielsen’s model (1992)


Nielsen (1992), on the other hand, categorizes problems in three different ways: by severity (i.e., expected impact on the users), by heuristic, and by the location in the dialogue. Following are the examples of each category.

Severity of problem: Usability problems causing a major impact, and usability problems causing a minor impact on the users.

Applicable heuristic: Simple and natural dialogue, speak the user’s language, minimize user memory load, be consistent, provide feedback, provide clearly marked exits, provide shortcuts, good error messages, prevent errors, etc.

Where is problem located: A single dialogue element, comparison of two elements, the overall structure of dialogue, Something missing, etc.


Strength and Weakness

Unlike the Dumas and Redish (1999) model, Nielsen’s model accounts for scope (location), severity and design heuristics, thus, making it easy to recognize and organize problems. However, Nielsen did point out that practitioners with a higher level of expertise in a particular kind of interface are more likely to recognize usability problems, especially those that are unique to that interface. He also mentioned that a group of practitioners/specialists working together uncover more usability problems as opposed to an individual specialist. The fact that the model is restricted to a certain level of expertise and to a group of practitioners, can be seen as a weakness of this model.


3. The NCPI model


Usability is an evolving field. Even though the field has gained popularity in recent years, not many people understand its concepts, principles, impact, and benefits. It, therefore, becomes imperative to organize and report usability problems in a way that is comprehensible to everyone. Rather than using Dumas and Redish’s scope method or Nielsen’s applicable heuristic and location method, I prefer categorizing problems into groups that are easy to comprehend by non-usability practitioners, especially stakeholders, and clients. The groups are as follows:

(N) Navigation: This category includes all the problems that restrict the user from navigating from one location to another within the product. Apart from the common navigational components like menu, breadcrumbs, footer links, quick links, etc.; navigation also includes the overall information architecture.

(C) Content: This category focuses on the information value of the content, as opposed to the presentation of it. The content topics include editorial style, internationalization, language, and accessibility.

(P) Presentation: This category refers to all the problems related to layout, aesthetics, information hierarchy, visual hierarchy, alignment and positioning of elements.

(I) Interaction: Problems associated with animation, transitions and actionable items fall into this category.


Strength and Weakness

These NCPI categories not only help non-usability practitioners to understand and acknowledge usability problems but also acts as a guide, a framework, for novice usability practitioners to identify and uncover problems.



Prioritization of usability problems


Once the usability problems have been identified, it is important to prioritize them in order to decide the sequence in which they will be tackled. There are various ways of Prioritizing usability problems.


1. Donahue (2001) - Cost-Benefit analysis


(Donahue, 2001) After identifying usability problems, costs associated with the correction or redesign of the system is calculated. Along with the cost, the benefit of the redesign is also calculated. Once the cost for benefit analysis for each usability problem has been arrived upon, problems can be prioritized based on the ROI (Donahue, 2001).


Strength and Weakness

The method gives a clear cost-benefit ratio for each problem making it easy for stakeholders, clients, and other non-usability practitioners to make prioritization decisions. However, it has a huge disadvantage associated with it. Quantifying the benefits of improved usability is a great challenge and requires a high level of expertise and experience.


2. Rubin and Chisnell (2008) - Four-level identifier


The severity model proposed by Rubin and Chisnell (2008) is a four-level identifier similar to the one proposed by Dumas and Redish (1999) that was discussed earlier in the paper. However, in Rubin and Chisnell model, Level 4 represents the most severe problems as opposed to Level 1 being the most severe in Dumas and Redish model. Also, Rubin and Chisnell severity model takes into account the probability that the problem will be encountered. The model assigns a criticality rating to the problems by combining the severity and occurrence probability of each problem using the formula shown below.

Criticality = Severity + Probability of Occurrence


Strength and Weakness

The emphasis on the probability and severity, but, neglects the impact or scope of the problem while calculating the criticality of a problem. It would make more sense to combine all the three aspects (scope, probability, and severity) in order to prioritize usability problems.


3. Nielsen’s Severity ranking (1995)


(Nielsen, 1995) A prioritization model proposed by Nielsen incorporates the above-discussed aspects (severity, probability, and scope) of a problem. He, however, describes the three factors as frequency (common or rare), impact (how easy is it for the user to overcome the problem?), and persistence (can users learn to avoid it or is it persistent?). Combining these three factors gives a prioritization scheme as shown in the list below.

Level 0 - Not a usability problem at all

Level 1 - Cosmetic problem, fix only if there is extra time.

Level 2 - Minor usability problem. Low priority fix.

Level 3 - Major usability problem. Important to fix. High priority.

Level 4 - Usability catastrophe. Imperative to fix before the product is released.


Strength and Weakness

Nielsen did not give a fixed method for calculating the total severity score for a given problem. His method relies on the practitioner’s judgment, making this scheme relative and not accurate. Also, all the prioritization schemes discussed so far are user-centric and does not take business value/proposition into consideration. Thus, arises a need for more accurate schemes that also align with the business.


4. HP Scale Model


The scale model used at HP is a good epitome for such scheme. It takes in account the following four factors: severity (impact on task accomplishment), commonality (number of users that will encounter the issue), frequency (number of times the issue surfaces), and business importance (impact on key dimensions of the business). A business priority scale is arrived by combining the four factors using the below formula.

BPI = Severity x (Commonality + Frequency) x Business Importance


Strength and Weakness

Though the scale seems complex, it drives priority keeping four critical aspects in consideration and assigns a number to each problem making it easy for stakeholders, client, and other business people to understand. As science writer Margaret Wertheim and many other mathematicians say “numbers speak louder than words”.



Conclusion


Now that we have discussed strengths and weaknesses of various categorization and prioritization schemes, it’ll be easy to make the choices. For categorization, I’d like to use the NCPI model as it not only enables stakeholders, and clients to understand the problem; but also, acts as a framework for presenting and simultaneously educating them. For prioritization, I’d like to use the HP scale model for projects that involve stakeholders/clients who rely more on numbers. And for the more creative audience, I’d like to use Nielsen’s severity ranking scheme keeping the complex calculations aside.

In closing, organizing or categorizing usability problems is an important step in any testing/evaluating process. It is up to the testers and design teams to choose which scheme(s) work best for them as well as their audience. As timelines become short, simple and effective processes such as those mentioned in this paper will provide useful guidelines to categorize and prioritize usability problems.



References


Donahue, G. (2001). Usability and the bottom line. IEEE Software, 18(1), 31-37. Retrieved from http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=903161

Dumas, J., & Redish, J. (1999). A Practical Guide to Usability Testing (pp. 322-326). Intellect Ltd.

Nielsen, J. (1995). Severity ratings for usability problems. Papers and Essays, 54, 1-2.

Nielsen, J. (1992, June). Finding usability problems through heuristic evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 373-380). ACM.

Rubin, J. & Chisnell, D. (2008). Handbook of usability testing: How to plan, design and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley Publishing, Inc.

Wilson, C. (2003). Defining, Categorizing, and Prioritizing Usability Problems.